[WIP] feat: add persistent custom operator registry by cmgzn · Pull Request #968 · datajuicer/data-juicer

cmgzn · 2026-04-15T03:56:12Z

Summary

Add a persistent JSON-based registry (~/.data_juicer/custom_op.json) so that user-defined custom operators survive across processes without requiring re-registration on every run.

Motivation

Previously, custom operators had to be re-loaded via config every time a process started. This made it cumbersome to work with reusable custom ops across sessions, scripts, and CLI invocations.

Changes

data_juicer/utils/custom_op.py (new) — Core module for persistent custom op management:
- JSON registry at ~/.data_juicer/custom_op.json storing source paths keyed by op name
- load_persistent_custom_ops() replays registrations on startup, auto-cleaning stale entries
- CLI interface: python -m data_juicer.utils.custom_op {list,register,unregister,reset}
- Dynamic module/package loading extracted from config.py
data_juicer/utils/registry.py — Add unregister_module() to Registry class
data_juicer/ops/__init__.py — Call load_persistent_custom_ops() at import time after built-in ops are loaded
data_juicer/config/config.py — Replace inline loading logic with a re-export from custom_op for backward compatibility
data_juicer/tools/op_search.py — Harden OPRecord to handle custom ops with non-standard module paths, missing source files, and absent test files
data_juicer/tools/DJ_mcp_granular_ops.py — Adapt MCP tooling for enhanced OPRecord fields
docs/DeveloperGuide.md, docs/DeveloperGuide_ZH.md — Document the new persistent registration workflow

Testing

tests/utils/test_custom_op.py
tests/tools/test_op_search.py

…tionality

gemini-code-assist

Code Review

This pull request introduces a persistent custom operator registry, enabling externally developed operators to be registered and automatically loaded across sessions. Key changes include the addition of data_juicer.utils.custom_op for registry management, an enhanced CLI for the operator search tool, and improved parameter handling in MCP tool generation. Review feedback identifies safety concerns regarding the manual cleanup of sys.modules during operator unregistration and highlights inconsistencies between the documentation and implementation regarding registry filenames and environment variables. Additionally, more robust error handling was recommended for resolving operator source paths.

ShenQianli · 2026-04-15T10:20:57Z

+            except Exception as e:
+                # Clean up partially-initialized module to avoid stale entries
+                sys.modules.pop(module_name, None)
+                raise RuntimeError(f"Error loading '{abs_path}' as '{module_name}': {e}")


check if we need to rollback OPERATORS here

Added _rollback_operators() helper. Both file and package loading branches now snapshot OPERATORS before loading and roll back on failure.

ShenQianli

the persistence model is at the operator level, but the real loading unit is the module/package path -> unreliable for package-based custom ops, relative imports, and multi-operator modules -> i think persistence should be path-based instead.

… instead of operator names

…m-operators

… function

- Remove redundant _read_registry() call in stale entry cleanup (race condition) - Restore single-quote style in __all__ to avoid unrelated formatting changes - Add missing blank lines between test classes (PEP 8)

… custom operations loading and testing isolation.

cmgzn · 2026-05-18T07:29:35Z

the persistence model is at the operator level, but the real loading unit is the module/package path -> unreliable for package-based custom ops, relative imports, and multi-operator modules -> i think persistence should be path-based instead.

The registry is now keyed by registration path (file/directory), not operator name. Operator names are always derived at runtime from the live OPERATORS registry after loading. This handles package-based custom ops, relative imports, and multi-operator modules correctly.

cmgzn added 4 commits April 15, 2026 11:07

feat: add persistent custom operator registration functionality

acad2d8

refactor: simplify custom operator loading logic

0e6a216

refactor: enhance signature annotation resolution for ops search func…

de1e574

…tionality

refactor: rename custom op registry file path

63d53f3

cmgzn requested review from HYLcool, ShenQianli and yxdyc April 15, 2026 03:56

cmgzn had a problem deploying to Testing April 15, 2026 03:56 — with GitHub Actions Failure

yxdyc requested a review from cyruszhang April 15, 2026 03:56

docs: update registry path and env var names in Developer Guide

ea55f7d

cmgzn had a problem deploying to Testing April 15, 2026 03:58 — with GitHub Actions Failure

gemini-code-assist Bot reviewed Apr 15, 2026

View reviewed changes

Comment thread data_juicer/utils/custom_op.py Outdated

Comment thread data_juicer/utils/custom_op.py Outdated

Comment thread docs/DeveloperGuide.md

Comment thread docs/DeveloperGuide_ZH.md

Comment thread data_juicer/tools/op_search.py Outdated

fix: enhance error handling for op search paths

18b437d

cmgzn had a problem deploying to Testing April 15, 2026 04:07 — with GitHub Actions Failure

correct typo and improve error handling for custom operators loading

56d6fb4

cmgzn had a problem deploying to Testing April 15, 2026 05:27 — with GitHub Actions Failure

ShenQianli reviewed Apr 15, 2026

View reviewed changes

ShenQianli requested changes Apr 15, 2026

View reviewed changes

cmgzn had a problem deploying to Testing April 16, 2026 07:19 — with GitHub Actions Failure

cmgzn force-pushed the feat/persistent-custom-operators branch from 2d1c83c to 21f6185 Compare April 16, 2026 08:13

cmgzn had a problem deploying to Testing April 16, 2026 08:13 — with GitHub Actions Failure

refactor: update custom operator management to use registration paths…

1e075cd

… instead of operator names

cmgzn force-pushed the feat/persistent-custom-operators branch from 21f6185 to 1e075cd Compare April 16, 2026 09:36

cmgzn had a problem deploying to Testing April 16, 2026 09:37 — with GitHub Actions Failure

cmgzn temporarily deployed to Testing April 16, 2026 09:37 — with GitHub Actions Inactive

Merge remote-tracking branch 'origin/main' into feat/persistent-custo…

1a08f00

…m-operators

cmgzn had a problem deploying to Testing April 23, 2026 03:50 — with GitHub Actions Failure

cmgzn changed the title ~~feat: add persistent custom operator registry~~ [WIP] feat: add persistent custom operator registry Apr 23, 2026

Merge remote-tracking branch 'origin/main' into feat/persistent-custo…

db23626

…m-operators

cmgzn had a problem deploying to Testing May 7, 2026 08:50 — with GitHub Actions Error

fix(registry): update registry version from 2 to 1 in _empty_registry…

6828fca

… function

cmgzn force-pushed the feat/persistent-custom-operators branch from ca78d61 to 6828fca Compare May 7, 2026 09:00

cmgzn temporarily deployed to Testing May 7, 2026 09:00 — with GitHub Actions Inactive

cmgzn had a problem deploying to Testing May 7, 2026 09:00 — with GitHub Actions Failure

cmgzn had a problem deploying to Testing May 9, 2026 08:39 — with GitHub Actions Failure

ci: add condition for forked PRs in unit test workflow

1b24d06

cmgzn temporarily deployed to Testing May 9, 2026 09:40 — with GitHub Actions Inactive

cmgzn added 3 commits May 15, 2026 10:54

Merge branch 'main' into feat/persistent-custom-operators

eb5c854

mini fix

9bc510e

- Remove redundant _read_registry() call in stale entry cleanup (race condition) - Restore single-quote style in __all__ to avoid unrelated formatting changes - Add missing blank lines between test classes (PEP 8)

refactor: improve error handling and add shared test base classes for…

4422673

… custom operations loading and testing isolation.

cmgzn requested a deployment to Testing May 18, 2026 07:09 — with GitHub Actions Waiting

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] feat: add persistent custom operator registry#968

[WIP] feat: add persistent custom operator registry#968
cmgzn wants to merge 15 commits into
datajuicer:mainfrom
cmgzn:feat/persistent-custom-operators

cmgzn commented Apr 15, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ShenQianli Apr 15, 2026

Uh oh!

cmgzn May 18, 2026

Uh oh!

ShenQianli left a comment

Uh oh!

cmgzn commented May 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

cmgzn commented Apr 15, 2026

Summary

Motivation

Changes

Testing

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ShenQianli Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

cmgzn May 18, 2026

Choose a reason for hiding this comment

Uh oh!

ShenQianli left a comment

Choose a reason for hiding this comment

Uh oh!

cmgzn commented May 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants